Database products such as the MySQL open source database are used by increasing numbers of organizations worldwide. With an estimated fifty thousand downloads per day, the MySQL database is a popular choice for database developers, administrators (DBAs), and IT managers who want a high-performance database that is reliable, affordable, and easy to use. For some organizations, such databases can become major production environments, and are used to run applications that are core to business operations. As their business grows, so does the need for complex data management. Many of these organizations would like to migrate data from, e.g. their MySQL database, to other commercial databases such as Oracle/DB2/SqlServer, for scalability reasons.
In addition to data migrations, some organizations who use MySQL as their original database may also want to integrate their data with other operational systems and databases. An example may be integration with real time data warehousing systems or other financial applications that runs on different database systems. Such organizations often require a solution that provides heterogeneous replication with low latency for zero downtime migrations.
It is also sometimes the case that an organization would prefer not to burden a production server for frequent reporting queries, since a great number of read-only queries for reporting might slow down the performance of production server. Some organizations may prefer to offload the report-related query processing to another, perhaps lesser-performing, non-production database. For example, some organizations may want to use a first type of database server as their reporting server, with, e.g. MySQL running either in Linux or Windows OS; and a second type of database server, e.g. Oracle running as a high-performance production server.
To enable integration of data persisted in a MySQL database with other non-MySQL systems, custom programs or software components such as ETL/EAI/EII (via a gateway or additional products) can be used. A problem with these technologies is that they are inefficient at processing non-bulk data (ETL), require application modification to publish the data from the application (EAI), or require application access to the MySQL system and the non-MySQL systems and integrate the data as needed. At high volumes, it is computationally taxing to run ETL continuously to integrate the changed data, EAI integration requires modification/access to the application thereby affecting application response times, and EII is slow since it involves access to multiple systems over different (remote) networks.
Accordingly, to date there is no log-based change data capture (CDC) program to perform logical replication of, e.g. MySQL transactional data from a MySQL database to a non-MySQL database, other than periodic (nightly) ETL, EAI, or EII ways of dealing with such integration/synchronization. These are the general areas that embodiments of the invention are intended to address.